In Search of an Entity Resolution OASIS: Optimal Asymptotic Sequential Importance Sampling

Entity resolution (ER) presents unique challenges for evaluation methodology. While crowd sourcing provides a platform to acquire ground truth, sound approaches to sampling must drive labelling efforts. In ER, extreme class imbalance between matching and non-matching records can lead to enormous labelling requirements when seeking statistically consistent estimates of population parameters. This paper addresses this important challenge with the OASIS algorithm. OASIS draws samples from a (biased) instrumental distribution, chosen to have optimal asymptotic variance. As new labels are collected OASIS updates this instrumental distribution via a Bayesian latent variable model of the annotator oracle, to quickly focus on regions providing more information. We prove that resulting estimates of F-measure, precision, recall converge to the true population values. Thorough comparisons of sampling methods on a variety of ER datasets demonstrate significant labelling reductions of up to 75% without loss to estimate accuracy.